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Abstract Body 

Background/context: 

This paper presents final findings from the Reading First Impact Study, a congressionally 
mandated evaluation of the federal government’s $1.0 billion-per-year initiative to help all 
children read at or above grade level by the end of third grade. The No Child Left Behind Act of 
2001 (PL 107-110, Title I, Part B, Subpart 1) established Reading First (RF) and mandated its 
evaluation. The Reading First legislation requires programs and instruction to be based on 
scientific research in reading, and aims to ensure that all children can read at or above grade level 
by the end of third grade, thereby significantly reducing the number of students who experience 
difficulties in later years. The overarching goal of Reading First is to improve students’ reading 
achievement. In September 2003, the US Department of Education contracted with Abt 
Associates to design and conduct the Reading First Impact Study (RFIS). 

Purpose/objective/research question/focus of study: 

The Reading First Program is a central element of the No Child Left Behind legislation (No 
Child Left Behind Act, 2001). It builds on findings reported in a national consensus report 
(NICHD, 2000), about proven strategies to reduce the prevalence of reading difficulty in the 
early grades in order to help children read at or above grade level by the time they reach third 
grade. The program provides resources to schools for professional development, for purchasing 
reading programs/materials, for assessments of students, and for salaries of reading coaches who 
can help classroom teachers. Reading First departs from historical precedent in its reach into the 
content of instruction. Traditionally, the federal education agency has been focused on issues of 
access, and determination of actual curriculum content has been left to states and localities. The 
legislation contained a savvy blend of federal guidance for local decision-making; it did not 
endorse any reading materials or professional development or assessments, but rather articulated 
the procedures states and districts were required to use as they made local decisions about which 
districts and schools to fund as well as which specific tools those districts and schools selected 
help to implement the program. The Reading First Impact Study was designed to assess the 
impact of the Reading First Program on two core domains: classroom reading instruction and 
student reading achievement. This paper will summarize the methods used and the study’s 
findings on the two questions of program impact. 



Setting: 

The RF program targets low-income, low-performing elementary schools. The schools are 
situated in states (and districts) that prepared detailed plans for increasing the use of teachers’ 
research-based instruction through intensive professional development for teachers, reading 
coaches, and administrators, with the explicit aim of reaching out to all eligible schools over time 
(No Child Left Behind Act, 2001). 

Population/Participants/Subj ects : 

The study sample includes low-income, low performing schools that were rated as eligible to 
receive RF funding; half of the study schools received RF funding and the other half did not. The 
sample was chosen purposefully to meet the requirements of the regression discontinuity design 
(described below). Of the 28 school districts and one state RF program that met the criteria, 16 
districts and the state program participated in the regression discontinuity design. The final 
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study sample reflected wide variation in district characteristics and provided enough schools to 
meet the study’s sample size requirements. One other school district agreed to randomly assign 
some of its eligible schools to RF or a control group. The final sample includes 248 schools 
across these 18 sites. 

Intervention/Program/Practice : 

The Reading First program’s overarching goal is to improve the quality of reading instruction — 
and thereby improve the reading skills and achievement of children in kindergarten through third 
grade — by implementing the use of research-based instruction and materials. Reading First aims 
to increase both the availability and quality of professional development for all K-3 teachers, 
including special education teachers, so that they have the necessary skills to teach research- 
based reading programs effectively An important provision of the RF legislation is that 
professional development be available to staff in all schools, not only those with RF funding. 
Specifically, K-3 teachers are eligible to participate in professional development paid for by 
district RF funds, and K-12 Special Education teachers are eligible to participate in professional 
development paid for by state RF funds. 



Reading programs and instruction methods should incorporate the five essential elements of 
effective primary grade reading instruction, specified in the legislation: 1) phonemic awareness; 
2) phonics; 3) vocabulary development; 4) reading fluency, including oral reading skills; and 5) 
reading comprehension strategies. Reading First also emphasizes the use of assessments, both to 
monitor progress and to identify and address students’ reading problems early, by helping 
classroom teachers to screen for, identify, and overcome barriers to students’ ability to read at 
grade level by the end of third grade. 



Research Design: 

The RFIS employs a regression discontinuity design that capitalizes on the systematic process 
used by a number of school districts to allocate their RF funds. A regression discontinuity design 
is the strongest quasi-experimental method that exists for estimating program impacts. Under the 
certain conditions outlined below, all of which are met by the present study, this method can 
produce unbiased estimates of program impacts: 1) Eligible schools were rank-ordered for 
funding based on a quantitative rating, such as an indicator of past student reading performance 
or poverty; 2) A cut-point in the rank-ordered priority list separated schools that did or did not 
receive RF grants, and this cut-point was set without knowing which schools would then receive 
funding; and 3) funding decisions were based only on whether a school’s rating was above or 
below its local cut-point; nothing superseded these decisions. 

Additionally, the functional form of the relationship between schools’ ratings and outcomes can 
be modeled. Under these conditions, there should be no systematic differences between eligible 
schools that received a RF grant (RF schools) and eligible schools that did not (non-RF schools), 
except for the characteristics associated with the school rating used to determine the funding 
decision. By controlling for differences in schools’ ratings, one can then control statistically for 
all systematic pre-existing differences between the two groups. This makes it possible to estimate 
the impact of RF by comparing the outcomes for RF schools and non-RF schools in the study 
sample, controlling for differences in their ratings. 
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Data Collection and Analysis: 

The RFIS is a multi-year study, with data collection over three school years, 2004-05, 2005-06, 
and 2006-07 (See Exhibit 1). The study collected several different types of data to address its 
evaluation questions, including direct assessment of students, classroom observations, and 
surveys of principals, teachers and reading coaches. These data were used to measure students’ 
reading comprehension and decoding skills; teachers’ instructional practices in reading; and 
aspects of scientifically based reading instruction such as professional development in 
scientifically based reading instruction, amount of reading instruction, supports for struggling 
readers, and use of assessments. This paper presents impact estimates for these measures, 
described more fully below. 

Student reading performance was assessed with the reading comprehension sub test of the 
Stanford Achievement Test, 10th Edition (SAT 10, Harcourt Assessment, Inc., 2004). The SAT 
10 was administered to students in grades one, two and three during fall 2004, spring 2005, 
spring 2006, and spring 2007. In the spring of 2007 only, first grade students were assessed with 
the Test of Silent Word Reading Fluency (TOSWRF, Mather et al., 2004), a measure designed to 
assess students’ ability to decode words from among strings of letters. 

Classroom reading instruction was assessed in first-grade and second-grade reading classes 
through an observation system developed by the study team called the Instructional Practice in 
Reading Inventory (IPRI). Observations were conducted during scheduled reading blocks in each 
sampled classroom on two consecutive days during each wave of data collection: spring 2005, 
fall 2005 and spring 2006, and fall 2006 and spring 2007. The IPRI, which is designed to record 
instructional behaviors in a series of three-minute intervals, can be used for observations of 
varying lengths, reflecting the fact that schools’ defined reading blocks can and do vary. Most 
reading blocks are 90 minutes or more. Eight outcome measures of classroom reading instruction 
were created from IPRI data to represent the components of reading instruction emphasized by 
the RF legislation. Six of these measures are reported in terms of the amount of time spent on the 
various dimensions of instruction. Two of these measures are reported in terms of the proportion 
of the intervals within each observation. 

Professional development in scientifically based reading instruction, amount of reading 
instruction, supports for struggling readers, and use of assessments. Within these four 
domains, eight outcome measures were created based on data from surveys of principals, reading 
coaches, and teachers about school and classroom resources. The eight outcome measures 
represent aspects of scientifically based reading instruction promoted in the RF legislation and 
guidance. Surveys were fielded in spring 2005 and again in spring 2007; findings presented here 
are from 2007 surveys only. 

The RFIS is comprised of separate regression discontinuity designs for each of the 17 RDD sites, 
plus a group-randomized experiment for the experimental site; as a result, the impact estimates 
presented are averaged across the study’s 18 sites. The average is weighted in proportion to the 
number of RF schools in the study sample from each site. Findings presented in this report 
therefore represent average impacts for the typical RF school in the sample. For each measure, 
an extension of the statistical model below was used to estimate the impacts of RF in the 17 
RDD sites: 
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( 1 ) 



Yk — flo + f3 1 Tk + y 3 2 Rk + jLik 



where: 



Yk = the outcome measure for school k, 

T k = one if school k is a RF school and zero otherwise, 

R k = the value of the rating for school k, 

p k = a random error for school k that is assumed to be independently and 
identically distributed. 



The coefficient EF summarizes the continuous relationship between outcomes and ratings that 
exists on either side of the cut-point. Controlling for this relationship controls for all systematic 
pre-existing differences between RF schools and non-RF schools. The coefficient, Bj, for the 
treatment indicator, T k , provides an estimate of the impact of RF. To increase the precision of 
impact estimates a limited number of covariates (student background characteristics and/or 
school baseline test scores) were added to the estimation model. In addition, because of 
clustering (for example, students within classrooms, and classrooms within schools), multi-level 
models were used to estimate impacts on all outcomes. 

Findings/Results: 

Exhibit 2 reports average impacts on classroom reading instruction and student reading 
comprehension pooled across school years 2004-05 and 2005-06 and 2006-07. Exhibit 3 reports 
average impacts on key components of scientifically based reading instruction from spring 2007. 
Exhibit 4 reports the average impact on first graders’ decoding skills from spring 2007. Impacts 
were estimated for each study site and averaged across sites in proportion to their number of RF 
schools in the sample. Average impacts thus represent the typical study school. Key findings are 
as follows: 



• Reading First produced a positive and statistically significant impact on amount of 
instructional time spent on the five essential components of reading instruction promoted 
by the program (phonemic awareness, phonics, vocabulary, fluency, and comprehension) 
in grades one and two. The impact was equivalent to an effect size of 0.33 standard 
deviations in grade one and 0.46 standard deviations in grade two. 

• Reading First produced positive and statistically significant impacts on multiple practices 
that are promoted by the program, including professional development in scientifically 
based reading instruction (SBRI), support from full-time reading coaches, amount of 
reading instruction, and supports available for struggling readers. 

• Reading First did not produce a statistically significant impact on student reading 
comprehension test scores in grades one, two or three. 

• Reading First produced a positive and statistically significant impact on decoding among 
first grade students tested in one school year (spring 2007). The impact was equivalent to 
an effect size of 0.17 standard deviations. Because the test of students’ decoding skills 
was only administered in a single grade and a single year, it is not possible to provide an 
estimate of Reading First’s overall impact on decoding skills across multiple grades and 
across all three years of data collection, as was done for reading comprehension. 
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